Method to mark and exploit at least one sequence record of a video presentation

ABSTRACT

The present invention proposes a method to mark and exploit at least one sequence record of a video presentation played on a multimedia unit, said method comprising the steps of: during the video presentation, receiving a command from a user to mark a currently displayed video sequence, said command initiating the step of: creating a sequence record comprising a time index or frame index, allowing to locate the proper part of the video presentation, and a reference of the video presentation. At a later stage, requesting the edition of the sequence record by: adding textual information which corresponds to the actual sequence, and storing the sequence record.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is continuation of U.S. patent application Ser. No.15/053,895 filed Feb. 25, 2016, which is a continuation of U.S. patentapplication Ser. No. 13/991,254 filed Jun. 3, 2013 (now U.S. Pat. No.9,307,185 issued Apr. 5, 2016), which is a U.S. National StageApplication of International Application No. PCT/EP2011/071492 filedDec. 1, 2011, which claims priority from US Provisional PatentApplication No. 61/418,925 filed Dec. 2, 2010. The entirety of all theabove-listed applications are incorporated herein by reference

INTRODUCTION

The invention refers to a Television system engaging the TV viewer in anenhanced interaction with the video content. The invention furtherrelies on a collaborative behaviour of TV viewers who activelyparticipate in the annotation of video assets in order to provideinformation to future viewers.

Problem to be Solved

When watching a movie or any video sequence, one may want toannotate/tag a particular image or sequence of images at some point inthe video.

The tag may be used as a bookmark to further jump from place to placealong the video stream or it may be attached to an object (car, watch,tool . . . ), a person (actor, guest . . . ) or a geographical positionor landscape that is recognized by the TV viewer because he is familiarwith the place.

Such metadata attached to a video may be a personal tag which is keptprivate and used for subsequent viewings; alternatively, it may beshared with some friends or it may be public information which is sharedwith the whole community in order to provide valuable information tofuture viewers.

Unfortunately, this tagging activity may divert the attention of theviewer, especially when it is necessary to type some text with a remotecontrol, a keyboard, a virtual keyboard, a smartphone, a tablet . . . .

One solution is to postpone this activity during an advertisement orsimply at the end of the movie. Unfortunately, the TV viewer may haveforgotten the visual context which has triggered the wish to tag thevideo, especially if the viewer wishes to submit multiple tags along thewhole movie.

BRIEF DESCRIPTION OF THE INVENTION

The present invention proposes a method to exploit of at least onesequence record of a video presentation played on a multimedia unit,said method comprising the steps of:

-   -   during the video presentation, receiving a command from a user        to mark a currently displayed video sequence, said command        initiating the step of:    -   creating a sequence record comprising a time index or frame        index, allowing to locate the proper part of the video        presentation, and a reference of the video presentation,

At a later stage, requesting the edition of the sequence record by:

-   -   adding textual information which corresponds to the actual        sequence,    -   storing the sequence record.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be better understood thanks to the attacheddrawings in which:

the FIG. 1 illustrates the system in which the invention can beimplemented

the FIG. 2 illustrates the method of the invention with the delayedannotation.

DETAILED DESCRIPTION

The FIG. 1 depicts the end-to-end system in the special case where thedecoder is a PVR and the tagging point is defined with the remotecontrol. In this configuration, the video samples are stored on thelocal hard disk until the TV viewer finalizes the annotation. Thepre-tags may be stored locally but the finalized tag objects areuploaded to the back office server to be used by the whole community.

The FIG. 1 illustrates the system comprising a broadcaster 1 sendingvideo presentations to at least one decoder 2. The video presentationcomprises a reference allowing to identify one video presentation amongthe video presentations broadcasted by the broadcaster. In theillustrated embodiment, the decoder has an Internet connexion to shareits sequence records or to combine with sequence records from otherusers.

The FIG. 2 illustrate the method of the invention in which a videopresentation or video stream is flagged at a time t1 or t2 correspondingto the time of the creation of the sequence record. At the end of thevideo presentation, the user can add additional data to annotate thesequence record.

Let's first define what a tag consists of. (A tag is equivalent as asequence record)

A tag is made of at least:

-   -   the reference of the TV event or VoD movie to which it is        attached.    -   a time index or frame index, allowing to locate the proper part        of the TV event or VoD movie for example    -   a word or a short sentence which corresponds to the actual        annotation In addition the tag can alternatively further        comprises:    -   a tag category (bookmark, object, person, location, . . . )    -   an identifier which uniquely identifies the originator of the        tag (e.g. subscriber_id, pseudo, email address . . . )    -   a publication state indicating if the tag is a personal tag        which is kept private or if it is shared with some friends or if        it can be released to the whole community as public information.

The proposed solution for delayed tagging consists in first marking thevideo sequence when watching the video and then subsequently tagging thevideo sequence at a later time based on a video sample (or a set ofstill images) captured at marking time to remember the visual context.

Let's go through the various steps of the procedure.

Marking

When the user decides to tag a video at some point in time, he simplytriggers the marking by either pressing a button of the remote controlor a graphic button of an application running on a personal device likea tablet, a smartphone, a PC . . . .

This operation generates a pre-tag object which contains at least thefollowing information:

-   -   the reference of the TV event or VoD movie to which it is        attached.    -   a time index or frame index,    -   an address pointer to a video sample (or a set of still images)        around the marking time

Further information can be attached to the pre-tag, i.e.:

-   -   the identifier of the originator

This pre-tag object may be stored temporarily in the decoder or uploadedto the back office for remote storage.

Simultaneously, a video sample (or a set of still images) is captured.

2 technical solutions are proposed to implement the acquisition of videosamples around the marking time:

1—in a first embodiment, the decoder is a Personal Video Recorderequipped with a mass storage capacity like a Hard Disk and the TV vieweruses his remote control to mark a tagging point. In the decoder, acircular buffer of pre-defined size (corresponding to a few seconds) iscontinuously filled with the video stream. When the user places a mark,the buffer is further filled for half of its total size; then it isfetched and stored for subsequent viewing. The address pointer to thestored buffer is added in the pre-tag object for later access.

2—in a second embodiment, the decoder is not necessarily a PVR decoderand it has limited storage capacity. Therefore, the video sample isstored at the head-end based on a notification of the TV viewer whoindicates a marking point. This notification can be made by the remotecontrol through the decoder if it has a return path or from any portabledevice connected to the Internet like a PC, a tablet or a smartphone.

Several implementations are possible at the Head-End: a unique videosample may be copied for each TV viewer indicating a marking point;alternatively, to save memory, the start/end address pointers to aunique copy of the video asset are saved for further access (this optionis typically applicable for a VoD service or a catch-up TV service forwhich the video file remains easily accessible in the Head-End once ithas been broadcast).

Alternatively, in case of limited storage capacity or limited bandwidthon the broadband link, still images can be captured and stored every nframes rather than a video sample in order to further remember thecontext around the marking point.

The FIG. 2 depicts this pre-tagging step:

Tagging

When the user is available to finalize the tagging, he/she goes throughthe following steps:

-   -   he/she selects a tagging session in his/her list of tagging        sessions    -   he/she selects a pre-tag in his/her list of pre-tags    -   he/she accesses the video sample captured at the related marking        time in order to remember the visual context.    -   he/she enters the tag i.e. a word or a short sentence    -   he/she optionally enters the tag category (bookmark, object,        person, location . . . )    -   he/she defines the publication state.

At this point, the tag is finalized. The video sample is erased to freethe memory, as well as the address pointer.

Publication

Once a tag is finalized, it is published according to its publicationstate. In most cases, it is simply uploaded in a back office server forsubsequent usage. It can also be stored locally in the terminal,especially if it is a private tag.

In addition, it can be explicitly published to one or several personsusing various communication systems such as emails, social networks(e.g. Facebook wall), SMS, micro-blogging (e.g. Twitter) . . . .

When all tags are stored in the back-office, it is important to recordthem efficiently in order to easily access them when requested. Forexample, all tags related to the same TV event/movie could be stored ina dedicated structure and ordered chronologically. More generally, anontology could be used to structure the tag's database.

Tag Usage

Assuming a movie has been tagged by one or several persons, the goal isnow to access the tags stored in the back office repository.

Several applications may be envisioned:

One option is to display all tags which have been associated to a movieby all contributors. This can be done in a graphical window of the TVscreen or on a portable device (smartphone, tablet, PC . . . ) that theTV viewer may hold. Obviously, the system shall make sure that the tagis displayed at the right time according to the corresponding timestamp. Optionally, it may be possible to display the tag a few secondsahead in order to warn the TV viewer in advance.

Alternatively, it may be possible to define some filters in order todisplay tags by category or by originator.

In a third option, it may be possible to use tags as bookmarks in orderto jump from tag to tag in a video sequence; this feature allows TVviewers to access key positions in the video sequence. Regarding theimplementation, the decoder uses the next tag time index received fromthe back office server in order to command the video server to jump tothe specified location in the stream.

This invention is integrated as a software package into the decoder ofthe user. This decoder or multimedia unit comprises a program memoryallowing to store program file and a processing device to execute thisprogram. The decoder also comprises a memory in the form of a hard driveor any non volatile memory to at least store locally the sequence recordwhile the video presentation is going on. The decoder can receive acommand from a remote control via infrared or wireless means. Thedecoder is in charge of receiving the video presentation, usually incompressed form (via various means such as Internet, cable, antenna,satellite) and process the signal (i.e. decompress or decrypt it in caseof conditional access video) and to pass it to the television. While thevideo signal is passed to the television, the decoder keeps track of anindex of the video presentation, index that allows to identify the videosequence currently displayed. This could be in the form of a time index,a frame index or an index (e.g. packet index) that allows to retrievelater the video sequence during which the user has activated the taggingfunction.

A predefined time can be set by the system or as a parameter accessibleby the user to move backward the video presentation and to store thisindex rather than the current index. The user can be interested to thefew second prior to the time he has pressed the command to tag the videopresentation. This predetermined time can be used to design the videobuffer used to fetch a sample of the video presentation. In an exemplaryembodiment, the predetermined time is 3 seconds, i.e. the decoder storeat least 3 seconds of the video already displayed in case that the userrequest a tag. In this case, the content of the buffer is stored in thehard drive of the decoder as past video event. It is possible that thefollowing 3 seconds are also added to this sample, having theconsequence that the 3 seconds before the user's command the 3 secondafter the user's command are store as video sample.

According to another embodiment, the video sample is added at the end ofthe video presentation, at the time the user enters into the mode ofcompleting the sequence record by additional data.

In case that the video presentation is provided by a broadcast servicethat allows retrieval of video presentation at a later stage, thedecoder does not need to buffer and store video sample at the time theuser requests a tag. The video sample can be added later with a requestsent to the broadcaster, this request containing at least the referenceof the video presentation and the index. This step of adding the videosample can be executed after the user has sent the sequence record tothe remote database. The latter can have access to the entire videopresentation and can obtain the video sample for storing it with theother data of the sequence record.

What is claimed is:
 1. A method for tagging a video presentation playedon a multimedia unit with at least one sequence record, said methodcomprising the steps of: during the playing of the video presentation,receiving a command, at the multimedia unit, from a user to mark acurrently displayed video sequence of the video presentation; creating,at the multimedia unit, a sequence record comprising a time index orframe index, and a reference of the video presentation, wherein the timeindex or frame index indicates a location of the marked portion of thevideo presentation; editing the sequence record of the videopresentation that comprises the time index or frame index, wherein theediting of the sequence record of the video presentation occurs at themultimedia unit and comprises: receiving from a user tag informationwhich corresponds to the sequence record of the video presentation;adding the tag information received from the user to the sequencerecord; and storing the edited sequence record of the video presentationthat contains the corresponding tag information received from the user.2. The method of claim 1, wherein editing the sequence record furthercomprises: adding a tag category to identify the type of video sequence.3. The method of claim 1, wherein the sequence record further comprisesan identifier of the user, said identifier being obtained automaticallyfrom the multimedia unit or being introduced during the editing thesequence record.
 4. The method of claim 1, wherein the sequence recordcomprises a publication state indicating if the record is: private; canbe shared within a specific group; or publicly available.
 5. The methodof claim 1, wherein the sequence record comprises a video sample of thetime corresponding to the index in the form of a still image or an videoextract.
 6. The method of claim 1, wherein editing the sequence recordfurther comprises saving the sequence record to a remote database. 7.The method of claim 6, further comprising: requesting one or more of thesequence records stored in the remote database; and displaying to theuser the sequence records.
 8. The method of claim 7, wherein the displayof the sequence records to the user is synchronized with the videopresentation using the time index or frame index contained in thesequence record.
 9. The method of claim 7, wherein the display of thesequence records to the user is synchronized in advance of time with thevideo presentation according to a predefined time.
 10. The method ofclaim 1, wherein the sequence record is edited during an advertisement.11. A decoder for tagging a video presentation, said decoder comprising:a video decoder configured to decompress a compressed video stream andoutput the decompressed video stream to a display unit; a processingdevice configured to execute a program; a first memory to store at leastthe program; a second memory for storing non-executable data; and aninput to receive a command from a user; wherein said decoder isconfigured to perform the following steps: during the playing of thevideo presentation, receiving a command from a user to mark a currentlydisplayed video sequence of the video presentation; creating a sequencerecord comprising a time index or frame index and a reference of thevideo presentation, wherein the time index or frame index indicates alocation of the marked portion of the video presentation; editing thesequence record of the video presentation that comprises the time indexor frame index, wherein the editing of the sequence record of the videopresentation comprises: receiving tag information which corresponds tothe edited sequence record of the video presentation from the user; andadding the tag information received from the user to the sequencerecord; and storing the edited sequence record of the video presentationthat contains the corresponding tag information.
 12. The decoder ofclaim 11, wherein editing the sequence record further comprises: addinga tag category to identify the type of video sequence.
 13. The decoderof claim 11, wherein the sequence record further comprises an identifierof the user, said identifier being obtained automatically from themultimedia unit or being introduced during the editing the sequencerecord.
 14. The decoder of claim 11, wherein the sequence recordcomprises a publication state indicating if the record is: private; canbe shared within a specific group; or publicly available.
 15. Thedecoder of claim 11, wherein the sequence record comprises a videosample of the time corresponding to the index in the form of a stillimage or an video extract.
 16. The decoder of claim 11, wherein editingthe sequence record further comprises saving the sequence record to aremote database.
 17. The decoder of claim 16, further configured toperform the steps of: requesting one or more of the sequence recordsstored in the remote database; and displaying to the user the sequencerecords.
 18. The decoder of claim 17, wherein the display of thesequence records to the user is synchronized with the video presentationusing the time index or frame index contained in the sequence record.19. The decoder of claim 17, wherein the display of the sequence recordsto the user is synchronized in advance of time with the videopresentation according to a predefined time.
 20. The decoder of claim11, wherein the editing of the sequence record occurs during anadvertisement.